The OS|periment

Hello, old blog friend.

Sorry for neglecting you lately. I don’t allocate as much time to OS research these days as the subject would warrant. I currently prefer to prioritize things like my precarious academic job and hobbies that fit better into the small time slices that I manage to allocate away from it, such as learning Rust and trying to build better Linux performance analysis tools with it.

I know it’s not an excuse, and I would like to get back to OS work at some point. Especially as every day remains a reminder of all that’s wrong about today’s operating systems and the abstractions that they present. And so today, I would like to discuss one particular OS abstraction that IMHO has not aged well and would be in need of a serious redesign in the next generation of operating systems: threads.

Continue reading →

A couple of weeks ago, I introduced you to one of my current pet projects, mostly aimed at making my next job easier but which could also be of interest to some of you: CLplusplus. Since then, that project has grown quite a bit, and I think it is now safe to say that it is about 50% there. Not in the usual terms of “50% of the code is written, untested and undocumented”, but in actual whole-project effort terms. Continue reading →

In my last post, I discussed my current attempts to build an emulator suitable for OS development. Today, I will discuss another project which I’m simultaneously working on, which is is more closely related to my next job than to OSdeving but could be of interest to some of you readers anyway: CLplusplus, a more modern C++ wrapper to the OpenCL API than what the OpenCL standard will provide.

Continue reading →

After spending some time hacking on Numerical Cookbook, I’ve reached enough confidence on my Ada skills to try out another, more ambitious project, which is more directly related to my long-term goal of resuming TOSP development. This week, said project has reached enough maturity for me to consider it camera-ready, so let me introduce you to my latest software experiment, EmulatorKit.

Continue reading →

Today, I would like to take this blog’s usual content aside for a second and express how outraged I feel about the attacks that are currently ongoing on the Protonmail encrypted email service.

As of today, end-to-end email encryption as a technology is not ready for the general public, because it is much too difficult to understand, setup and use. Services like Protonmail aim to change that and bring truly private telecommunications to the masses, and this is a cause for which I have the deepest respect.

Of course, I will gladly disagree with them on the technical level :

• Considering the amount of vulnerabilities that get disclosed in them per day, web browsers do not seem to be a safe platform for high-security applications.
• A centralized infrastructure is not a very good fit for a service that goes against the will of nation states, because it means there are only a few people to pressure/torture in order to get control of the infrastructure, and only a few computers to crash in order to bring the service down (as we are experiencing)

But the thing is, even if they are never going to build the perfect encrypted communication tool on their first try, Protonmail have got something running, it is ready right now (or at least it was until someone powerful decided to break it), and it is simple enough that nontechnical (or at least only mildly technical) people can use it easily. From a social point of view, that is something huge.

Thus, my dearest sympathies go to the Protonmail team in their current struggle to stay afloat. And so, dear reader, if you have the financial resources to do so, please go and help them. After being unfairly pressured into paying a huge ransom that turned out to be useless, they need all the financial support that they can get right now.

EDIT: And… they are back up ! Hope their anti-DDoS protection strategy will work.

Long-time readers of this blog will know that I have no fondness for the monolithic kernel design used by legacy operating systems. In addition to stuffing too much functionality into one piece of software, which is a failure at basic software design, and doing so on the single most privileged piece of code on a machine, which is a failure at basic software security, this design practice also has averse effects on performance by requiring frequent context switches between the OS kernel on one hand, and user-mode processes on the other hand.

In this blog post, I will highlight how this context switching overhead can be minimized through the use of more asynchronous system call and interprocess communication primitives. I will then show how these conclusions apply not only to traditional monolithic kernel designs, but also to more modern OS designs.

Continue reading →

Want some silly programmer humor ? Here’s a quick attempt at explaining how people would build a bicycle using the real-world equivalent of the programming languages I know of.

Continue reading →

Today, as another filler until I get enough time to resume OSdeving, I had some time to try out a quick performance test that has important implications for PC OSdeving : is it still possible, as of today, to get decent graphics performance out of software rendering in case a dedicated GPU driver is not available ? I’m not talking about displaying millions of triangles per second here, just displaying bitmap animations at a modern screen’s native refresh rate. Well, let’s find out !

Continue reading →

Hi everyone !

In case you are wondering, I’m not back working on TOSP just yet. My PhD has now entered its worst phase, with a dreadful last month of manuscript writing and an awfully ill-timed conference next week to make things worse. So I still cannot engage in a long-term personal programming project yet.

However, I have a little spare time left to code stuff I like at home. So I decided to practice my Ada a bit by translating some of Numerical Recipes‘ code snippets in Ada 2012, using my own coding style and a different name to avoid trademark lawsuits. I like it, it’s fun and full of bit-sized programming tasks that I can easily start and stop whenever I like. And at the same time, it also teaches a lot of enlightening stuff about how seemingly mundane decisions in programming language design, such as array bounds, can profoundly affect day-to-day coding in that language.

Which leads me to today’s thoughts on heap memory allocation and why I think we’re really overusing it these days.

Continue reading →

So, last November, I tried to resume blogging on OS-related matters, hoping that in time, I’d find enough resources to resume implementation too. Unfortunately, today, I have realized that I need to go in the opposite direction, and take another break from this project.

Why ? Because as a PhD student in my last year, I am extremely late at writing my thesis, and every work day starts to count. Yet simultaneously, outside of work, my life is getting busy again and I need more time to enjoy it (and get some sleep). Even if TOSP still matters a lot to me as a project, it cannot beat those more pressing concerns for now…

Anyhow, dear readers, it’s been a fun ride trying to get this project rolling for the second time. See you on my next attempt, perhaps!

Imagine that one day, as you come to the office, you would find a mysterious muscular man at the entrance, performing something that looks like an ID check. Mumbling about this world’s pandemic tendency towards security psychosis, you would search through your bag for your ID card or driver’s license. But as you would show it to him, he would say that he does not recognize it as valid.

Instead, he would direct you towards the services of Don Corleone, inc., selling *certified* IDs (basically a cheap copy of your id with a shiny stamp on it) for a hefty price, and say that he won’t let you enter without one of these.

Obviously, this scenario would be illegal in most countries, and you would be within your right to call the police. The state is the only legitimate authority trusted with the power to publish ID documents, and when it is democratically elected and subjected to the scrutinity of millions of citizens, one can expect that it won’t abuse its ID editing powers for fun and profit.

Yet for some reason, this is exactly the kind of protection racket that we deal with daily as we connect to the world wide web using HTTPS and have to interact with the current-generation Public Key Infrastructure (PKI), which is based on the concept of Certification Authorities (CA). And as I will elaborate, it gets worse. But first, let’s discuss why we are putting up with that. Continue reading →

As discussed in the last post, the purpose of authentication, using mechanisms such as passwords, is to ensure that the person in front of the keyboard is indeed the user she claims to be by testing her knowledge of some secret data. This is typically necessary in situations where users are not there to check that no one physically tampers with their computers, such as when a computer is left powered off, sleeping, or crunching numbers on its own with the screen locked.

Obviously, due to their sensitive nature, authentication secrets should be used sparingly. The more often they are used, the more likely they are to be intercepted by an attacker willing to perform identity theft, with potentially disastrous consequences. Yet I don’t know about you, but in a typical week, the two system dialogs which I interact with most often on my computers are these two fellows:

Both of them are dialogs with a simplistic appearance that any user application can replicate, which request me under a vague justification (“make changes”, “modify essential parts of [my] system”) to input a sensitive authentication credential giving full access to my system.

To Microsoft’s credit, their own implementation is a bit better, in that when setup properly, the dialog does not necessarily require a password to be input, appears less frequently, and can have a quite distinctive appearance (translucent black background) that is hard for malicious software to replicate. But I still think that these privilege elevation dialogs, and the habits they give users to input their (administrative) password whenever they are requested to without a clear justification, are a bit of a security issue. Continue reading →

Moving on in the present series regarding TOSP’s stance on security matters, I would like to stop discussing file access control and process sandboxing for a bit and discuss a somewhat higher-level, but equally important concept: authentication.

For those new to the term, here’s a short reminder: when you log into an operating system, much like on any other computer service, some process occurs where the operating system checks that you are the user who you claim to be. The OS does this by having you prove that you possess a piece of information which only you should know about, such as a password. This process is what is called user authentication.

In this post, I will discuss the various options which the modern world offers for authentication, their respective advantages and drawbacks, and the role which an operating system could play not only for the authentication of its own users, but also for authentication of users to third-party services.

Continue reading →

Ever since people decided, around the UNIX days, that file access needed to be controlled, all mainstream operating systems have featured file access control policies that follow the same basic principles.

Every user of a computer has a unique numerical identifier (UID), belongs to one or more groups each having a similarly unique identifier (GID), and somewhere up in the sky there is a “super-user”, called root or Administrator, that can bypass file access control entirely whenever the need arises. Processes are run under the responsibility of a given user and group, and files are also owned by a user and group, by default that of the process which created them.

Using this infrastructure, file access control has historically been implemented as a set of Read, Write and eXecute permissions that apply, in order of priority, to the user owning the file, to the group owning the file, and to everyone else. Then, as this design proved too limiting for complex organizations, an evolution of it called Access Control Lists or ACLs was implemented. In ACLs, file access control is still expressed in terms of users, groups, and RWX permissions, but can be composed of arbitrarily complex rules.

In this post, I will discuss why this security model has, in my view, outlived its usefulness in the area of general-purpose personal computing, and why I believe that process capabilities would be a better fit in this area. Continue reading →

Files are one of the most frequently used OS abstractions. They provide a convenient way for programs to produce long-lived output, and to store persistent state inbetween runs. From a user point of view, files also provide a standard, application-agnostic way to share any kind of data between computers, to be contrasted with the hoops that one has to jump through on those ill-designed mobile operating systems that do not have a user-visible file abstraction.

The power of files lies in how by using them, programs can store and retrieve named data without having to care about where and how it is stored. But such great power comes, as usual, with great responsibilities, which is why the abstraction can get pretty complex under the hood. After discussing a bit the UX of files earlier this year, I will now discuss the security implications of typical file manipulations. Continue reading →

As we’ve seen last week, a core security goal of a personal computer operating system is to ensure that programs only do what the computer users want. Unfortunately, not all software is equal on this front. Large, monolithic software with a broad feature set, such as modern web browsers, is for example hard to sandbox properly. In this blog, I will discuss an old software design trick from UNIX which can be used to reduce the need for such large software monoliths, and ways it can be made competitive with monolithic software from a performance point of view. Continue reading →

Ever since students have started playing pranks on the first academic computers, and malware has made its appearance, operating systems have needed to deal with security considerations. These days, one cannot just let random software do whatever it wants on personal computers, just like one couldn’t let users do whatever they want back in the days of multi-user timesharing systems.

As a middleman between software and hardware, the operating system is in a privileged position to police the wereabouts of software and users. But how exactly should it do that? Answering this question revolves around building an OS security policy. Who is the OS working for? What is it protecting? From which threats? And how?

This post will attempt to draft some security policy ideas for TOSP, by answering a number of these questions. It will be light on the “how”, however, which will be further discussed by subsequent articles. Continue reading →

Input sanitization is a well-accepted software design principle these days. Pretty much every developer worth his salt will agree that production-grade programs should not hang, crash, or do some very funky and undesirable things in response to user input, no matter how malicious the user is and how malformed the input is. But in low-level software development, when building programs that have a lot of power, I would argue that software should also be very mindful of its output, and the harm it may cause to the system they run on. This post will be a rough discussion of that. Continue reading →

Usability matters, for all software. Unless you have a reliable way to force users to use your most crappy software, you better make sure that they enjoy it. But it doesn’t have to be a chore, and can actually be a very fun and fulfilling challenge.

For example, contrary to popular belief, usability does not necessarily come at the expense of software power, but can actually enhance it, as usability helps users find out about the functionality that is available, and use it more efficiently through the reduction of error occurences and impact. And usability can improve the efficiency of security systems, by reducing the odds that users will try to get around these in potentially harmful ways, instead of understanding them and making the most of them.

Now, if we want to improve the usability of something as big and general-purpose as an operating system, how do we get around it? Unfortunately, there is no set of Big Rules Set In Stone that will make every software usable in a straightforward way. But, over time, a number of rules of thumb have proven quite effective at making software more pleasant to use, and this blog post will be about discussing these. Continue reading →

NOTE : This post was supposed to go online on the 19th, but remained mark as a draft in WordPress for some reason. Apologies for the unnecessary publication delay.

Operating systems are essentially about three things : abstracting out hardware, making unrelated application programs cooperate, and standardizing important functionality that is hard for application software to get right, such as GUI rendering or cryptography.

In current-generation OSs, the later mission is most frequently fulfilled by shared libraries, sometimes complemented by kernel code and user-mode processes when these prove absolutely necessary. In practice, however, the benefits of isolating system code into separate processes may be underrated. This post will discuss the relative merits of these two system services implementations, and why I believe that in many cases, the later option should receive more love than it currently does. Continue reading →

About 40 years ago, when the UNIX operating system was designed, its creators thought that a powerful virtual filesystem, shell scripts, and text I/O streams ought to be enough for everybody. And this is how Linux ended up nowadays with many incompatible way to do such fundamental things as boot-time initialization, GUI rendering, audio input and output, and now hardware accelerated graphics.

“Them fools !”, probably thought the Windows NT designers some 20 years later. “We shall make sure that every system functionality is specified in a standard way, so that this never happens to us.”. And nowadays, there are several incompatible APIs for pretty much every single functionality of Windows.

If there’s a lesson to be learned from the past there, I think it’s that operating systems need standard ways to do important things, but that doing standards right is unfortunately fiendishly hard. Continue reading →

Back when the UNIX operating system was designed, its development team set out to follow a set of design principles that became later known as the “Unix philosophy”. One of these design principles was that the UNIX software ecosystem should be planned as a set of small software modules that do one thing, do it well, and are easy to combine with one another in order to achieve a greater purpose. This approach is to be contrasted with the more common monolithic approach to software design, which attempts to build huge software product that address an entire business area on their own, such as Microsoft Visual Studio, or Adobe Photoshop, and the present post will be dedicated to exploring it. Continue reading →

After spending some time considering good practices for the handling and organization of data, let’s move a bit to the related subject of data loss and software reliability.

Most of the time, data loss occurs due to errors, a broad term which in software engineering jargon can designate two very different things:

1. A hardware, software or human issue requiring attention, which breaks the normal flow of computer programs.
2. A kind of user mistake where software does what a user ask for, but not what the user meant.

In this post, we will discuss what happens when errors are not managed properly, and software strategies for managing them by identifying, preventing, detecting, handling, and possibly reporting them. Continue reading →

To conclude the ongoing post series on data serialization and organization, I would like to introduce a proposal for a universal data serialization format. If you have studied the subject a bit, a knee-jerk reaction may be “Why? Don’t we already have JSON, CSV, XML YAML HDF5, and countless programming language-specific options?”. However, the huge amount of formats we have suggests to me that the data serialization format problem has not be fully solved. And I hope I can convince you that I have a sound proposal to solve it in a very general way, without going for extreme complexity at the same time. Continue reading →

This week, I had a very busy week-end, one of my storage drives went bad on me (yooohooo ata errors madness !), and I’ve been attenting the ESRF User meeting for professional networking reasons. All of this has eaten quite a bit into my weekly spare time, so there will be no post tomorrow. See you next week !

No human being could understand the source code of an even mildly complex computer program if it were displayed before his eyes in a fully inlined fashion, used unstructured data, and was stripped of any human-readable comment. For this reason, abstraction has always be a central concept in programming languages and practices.

Among popular programming abstrations, data encapsulation in particular has been used a lot in recent decades, in the wake of a persistent craze for the loosely defined concept of “object-oriented programming”. But is it such a good idea to decouple programs from the data they manipulate like that?

This article will discuss some issues of data encapsulation in the context of long-lived software like operating systems, and possible strategies to mitigate them. Continue reading →

So, in my last post, I have been discussing why I think that having a unified filesystem as a universal way to probe and locate system resources, like UNIX did, is a great thing. I also discussed why it appeared to me that UNIX and its successors designed it the wrong way, by using a hierarchical filesystem metaphor which turns out to be inappropriate when sorting out large amounts of data. I suggested that more clever use of file metadata might get around these problems, but didn’t expand on it yet. This will be the purpose of this post. Continue reading →

As discussed last week, a major role of an operating system is to manage shared computing resources. These resources can be pieces of data, or I/O streams, or procedural interfaces, or objects, in fact they can be any other kind of abstract computing resource.

I argued that the UNIX way of abstracting all of these as “files”, in the UNIX sense of a byte stream (preferably an ASCII text stream) with a variable amount of strange properties, was a questionable design choice, and discussed other ways to handle both pieces of data and I/O streams. But I agreed that they are both system resources, and that exposing them through a common interface is thus a good idea.

This post series will now focus on the role that the filesystem plays in such a unified exposure of system resources on UNIX systems, its limitations, and ways to improve upon it. Due to the complexity of the subject, it will take me a couple of blog posts to reach a satisfactory conclusion on this matter, so please bear with me if the discussion turns out a bit lengthy. Continue reading →

An early design decision of the UNIX operating system was that system resources were to be abstracted as files, plain text files whenever possible, organized in a hierarchical filesystem. Such resources would include pretty much everything that was shared between programs and other programs or users, from user data and software configuration to hardware interfaces.

At the time, this common resource access metaphor was a groundbreaking innovation, when competing operating systems would usually feature one unique interface per kind of system resource. The latter approach was obviously inferior, because its usability was much weaker (as one couldn’t reuse existing knowledge to face new problems), and it implied a much higher implementation complexity.

However, UNIX was designed in the 70s, and the computing landscape has changed dramatically since then. How did this design decision stand the test of time? This post will attempt to discuss its pros and cons in modern operating systems, and ways it could be improved upon. Continue reading →

Software interfaces rarely age well. Very often, products that looked fine enough in version 1 needed many major UI and API redesigns as they gained features and lost bugs, until they ended up unable to adapt further and either disappeared or underwent complete rewrites.

In the realm of OSdeving, this issue is particularly salient, as operating system software is extremely long-lived and has a lot of users and applications relying on its various interfaces, which means that changing any one of them is very difficult.

In this post, I will be discussing some ways OS interfaces can be made more future-proof. Continue reading →